{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Dealing with huge datasets\n", "\n", "Some data is too large for our small laptop. Some data is even too large for our terrabyte servers. Being smart about loading data can help us build better data science pipelines.\n", "\n", "As data science and machine learning become increasingly popular, the size of data sets used for analysis has grown exponentially. Sometimes, the data is so large that it cannot be loaded into memory on a single machine, even on a terabyte server. \n", "\n", "In such cases, loading data can become a bottleneck, hindering data analysis and decision-making. To overcome this challenge, data scientists must be innovative about loading data and building efficient data pipelines. Various techniques and tools are available to process and analyze large datasets, including parallel computing, distributed systems, cloud computing, and data streaming. \n", "\n", "This notebook will discuss the challenges of loading large datasets and explore some best practices for building efficient data science pipelines to handle big data. We will also explore popular tools and techniques for processing and analyzing large datasets." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How To" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "random_state = 42" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "df = pd.read_csv(\"data/housing.csv\")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
longitudelatitudehousing_median_agetotal_roomstotal_bedroomspopulationhouseholdsmedian_incomemedian_house_valueocean_proximity
0-122.2337.8841.0880.0129.0322.0126.08.3252452600.0NEAR BAY
1-122.2237.8621.07099.01106.02401.01138.08.3014358500.0NEAR BAY
2-122.2437.8552.01467.0190.0496.0177.07.2574352100.0NEAR BAY
3-122.2537.8552.01274.0235.0558.0219.05.6431341300.0NEAR BAY
4-122.2537.8552.01627.0280.0565.0259.03.8462342200.0NEAR BAY
\n", "
" ], "text/plain": [ " longitude latitude housing_median_age total_rooms total_bedrooms \\\n", "0 -122.23 37.88 41.0 880.0 129.0 \n", "1 -122.22 37.86 21.0 7099.0 1106.0 \n", "2 -122.24 37.85 52.0 1467.0 190.0 \n", "3 -122.25 37.85 52.0 1274.0 235.0 \n", "4 -122.25 37.85 52.0 1627.0 280.0 \n", "\n", " population households median_income median_house_value ocean_proximity \n", "0 322.0 126.0 8.3252 452600.0 NEAR BAY \n", "1 2401.0 1138.0 8.3014 358500.0 NEAR BAY \n", "2 496.0 177.0 7.2574 352100.0 NEAR BAY \n", "3 558.0 219.0 5.6431 341300.0 NEAR BAY \n", "4 565.0 259.0 3.8462 342200.0 NEAR BAY " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head(5)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index 128\n", "longitude 165120\n", "latitude 165120\n", "housing_median_age 165120\n", "total_rooms 165120\n", "total_bedrooms 165120\n", "population 165120\n", "households 165120\n", "median_income 165120\n", "median_house_value 165120\n", "ocean_proximity 1342940\n", "dtype: int64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.memory_usage(deep=True)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "longitude float64\n", "latitude float64\n", "housing_median_age float64\n", "total_rooms float64\n", "total_bedrooms float64\n", "population float64\n", "households float64\n", "median_income float64\n", "median_house_value float64\n", "ocean_proximity object\n", "dtype: object" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dtypes" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "df[\"ocean_proximity\"] = df[\"ocean_proximity\"].astype(\"category\")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "Index 128\n", "longitude 165120\n", "latitude 165120\n", "housing_median_age 165120\n", "total_rooms 165120\n", "total_bedrooms 165120\n", "population 165120\n", "households 165120\n", "median_income 165120\n", "median_house_value 165120\n", "ocean_proximity 21124\n", "dtype: int64" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.memory_usage(deep=True)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index 128\n", "longitude 165120\n", "latitude 165120\n", "housing_median_age 165120\n", "total_rooms 165120\n", "total_bedrooms 165120\n", "population 165120\n", "households 165120\n", "median_income 165120\n", "median_house_value 165120\n", "ocean_proximity 21124\n", "dtype: int64" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_small = pd.read_csv(\"data/housing.csv\", dtype={\"ocean_proximity\": \"category\"})\n", "df_small.memory_usage(deep=True)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
longitudelatitudeocean_proximity
0-122.2337.88NEAR BAY
1-122.2237.86NEAR BAY
2-122.2437.85NEAR BAY
3-122.2537.85NEAR BAY
4-122.2537.85NEAR BAY
\n", "
" ], "text/plain": [ " longitude latitude ocean_proximity\n", "0 -122.23 37.88 NEAR BAY\n", "1 -122.22 37.86 NEAR BAY\n", "2 -122.24 37.85 NEAR BAY\n", "3 -122.25 37.85 NEAR BAY\n", "4 -122.25 37.85 NEAR BAY" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_columns = pd.read_csv(\"data/housing.csv\", usecols=[\"longitude\", \"latitude\", \"ocean_proximity\"])\n", "df_columns.head()" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
longitudelatitudeocean_proximity
20046-119.0136.06INLAND
3024-119.4635.14INLAND
15663-122.4437.80NEAR BAY
20484-118.7234.28<1H OCEAN
9814-121.9336.62NEAR OCEAN
............
6052-117.7634.04INLAND
15975-122.4537.77NEAR BAY
14331-117.1532.72NEAR OCEAN
1606-122.0837.88NEAR BAY
10915-117.8733.73<1H OCEAN
\n", "

100 rows × 3 columns

\n", "
" ], "text/plain": [ " longitude latitude ocean_proximity\n", "20046 -119.01 36.06 INLAND\n", "3024 -119.46 35.14 INLAND\n", "15663 -122.44 37.80 NEAR BAY\n", "20484 -118.72 34.28 <1H OCEAN\n", "9814 -121.93 36.62 NEAR OCEAN\n", "... ... ... ...\n", "6052 -117.76 34.04 INLAND\n", "15975 -122.45 37.77 NEAR BAY\n", "14331 -117.15 32.72 NEAR OCEAN\n", "1606 -122.08 37.88 NEAR BAY\n", "10915 -117.87 33.73 <1H OCEAN\n", "\n", "[100 rows x 3 columns]" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_columns.sample(100, random_state=random_state)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise\n", "Check out the Dask playground for lazy dataframes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Additional Resources" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- [Scaling Pandas to Large Datasets](https://pandas.pydata.org/pandas-docs/stable/user_guide/scale.html)\n", "- [Dask](https://dask.org/)\n", "- [Making Pandas Fly](https://youtu.be/C1hqHk1SfrA?t=4551)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }